Geodesics of learned representations

We develop a new method for visualizing and refining the invariances of learned representations. Given two reference images (typically, differing by some transformation), we synthesize a sequence of images lying on a path between them that is of minimal length in the space of a representation (a “representational geodesic”). If the transformation relating the two reference images is an invariant of the representation, this sequence should follow the gradual evolution of this transformation. We use this method to assess the invariances of state-of-the-art image classification networks and find that surprisingly, they do not exhibit invariance to basic parametric transformations of translation, rotation, and dilation. Our method also suggests a remedy for these failures, and following this prescription, we show that the modified representation exhibits a high degree of invariance for a range of geometric image transformations.